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Abstract: This paper addresses the question when the underlying Markov 
process of a multiclass queueing network is positive Harris recurrent. It is 
well-known that stability of the fluid limit model is a sufficient condition for 
this. Hence, stability of fluid (limit) models is of vital interest. Recently, 
it has been shown that if the fluid model satisfies certain properties, it 
is stable if and only if there exists a Lyapunov function. In this paper a 
new method is provided to conclude that the underlying Markov process is 
positive Harris recurrent if the fluid model is stable by using explicitly the 
Lyapunov function of the fluid model. 
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1. Introduction 

Typically a multiclass queueing network consists of objects, for instance jobs 
or customers, that arc waiting for service in buffers in front of diverse stations. 
After service completion, a served job either moves to another buffer at some 
further station or leaves the network. 

A multiclass queueing network consists of J service stations and K classes 
of customers. The interarrival times for customers of class k G {1, ...K} are 
given by positive random variables afc(n), with n = 1,2,3,..., and the of class 
k customers are given by positive random variables Sfc(ro), with n = 1,2,3... . 
Each customer class is exclusively served at a certain station. The many-to-one 
mapping c : {1, K} — > {1, J} determines which customer class is served at 
which station. The corresponding J x K matrix O, called the constituency ma- 
trix, is defined by Cjk ■= 1 if c(k) = j and Cjk = else. For station j e {1, J}, 
the set C(j) :— {k g {1, K} : c(k) = j} is the collection of all customer classes 
that are served at station j. After a class k customer received service at the sta- 
tion c(fc) its routing is given by a if dimensional Bernoulli random variable cf> k . 
To be precise, each component of 4> h (n) is either or 1, but the entry 1 appears 
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at most once. Let e k denote the fcth standard basis vector for R K . Then, the nth 
served class k customer at station c(k) becomes a class I customer after service 
completion if <f> k (n) = e;, and the customer leaves the network if <fi k (n) = 0. 
For some customer class k the interarrival time may be afc(n) = oo for all n. 
Then, the exogenous arrival process is null. The corresponding notation is the 
following £ := {k e {1, ...,K} : ak(n) < oo, n > 1}. Further, the buffer at each 
station is assumed to have infinite capacity. Throughout this paper we pose the 
following general assumptions on the interarrival and the service times. 

(Al) The sequences ai, ax, si, sk and 1 , <j> K are identically and inde- 
pendently distributed and mutually independent. 
(A2) The first moments satisfy 



an 
/'/■■ 
Pk 



= E[afc(l)] 1 < oo for k £ £, 

= E[s fc (l)]- 1 <oofor ke{l,...,K}, 

= E[0 fe (l)] > for k e {1,...,K}, 



and the spectral radius of the matrix P = (Pi ... Pk) is strictly less than 
one. 

(A3) The distributions of the interarrival times are unbounded and spread out. 

A distribution v of the interarrival times is called unbounded if for each class 
k £ £ and for alH > it holds that 



,[a k (l)>t] :=^ C 



o fc (l)i/(d«) > 0. 



Unboundedness expresses that arbitrarily large interarrival times appear with 
positive probability. Moreover, the distribution of the interarrival times of cus- 
tomer class k e £ is said to spread out if there exists some l k € {1, 2, 3, ...} and 
some nonnegative function qk : R+ — > M+ with / °° qk(s)ds > such that for 
all < a < b, 



^2a k {i) £ [a,b] 



> 



q k {s) ds. 



We refer to the triple (a, s, </>) as the primitive increments of the multiclass 
qucucing network. 



State Space and underlying Markov process 

We restrict ourselves to head-of-the-line (HL) qucucing networks. The term HL 
indicates that within each class the customers are served in First-In-First-Out 
(FIFO) order. The evolution of the HL multiclass queueing network will be 
described by a stochastic process X = {X(t), t > 0}. The corresponding state 
space is denoted by (X, B), where B denotes the Borel c-algebra of X. The precise 
description depends on the particular discipline. We will not introduce the full 
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state space here, the interested reader is referred to [2, 6]. In general the state 
space is of the form 



Here (Z x R)°° denotes the set of finitely terminated sequences taking values in 
Zxl. The meaning of each of its components are the following. 

The global order of the customers in the network is given by the pair of 
sequences (k, w) = ((ki,Wi),(k2,W2),—,(ki,'Wi)) £ (Z x R)°°. The first entry 
ki £ {!,..., K{ denotes the current class of the ith customer and the second 
entry Wi > reflects the elapsed time since the customer i entered the class fcj. 
The order of elements (ki, in the sequence (k, w) is defined by the property 
that it is descending in Wi. The number of customers of each class is denoted 
by q = (qi, ■ ■ ■ ,qx) and \\q\\ := X)tLi Ik is the total number of customers in 
the state. The variable u £ denotes the residual interarrival time, where 
Uk > denotes the remaining time before the next arrival of a class k £ £ 
customer from outside the network. The component v £ M. K represents the 
residual service time, i.e. the coordinate Vk denotes the remaining time service 
time for the oldest class k customer, where Vk > and Vk — only if qk = 0. 
The component Zk of z £ [0, 1] K denotes the proportion of the service effort 
of station c(k) that the oldest class k customer receives, while other class k 
customers do not receive any service. This represents the HL property. For each 
station j we have that if YlkeCfj) 1 k > 0, it holds that YlkeC(j) z k = 1> where 
z k = if q k = 0. If station j is empty, i.e. J2keC(j) 9k = 0, then Y, k£C(j) z k = 0. 

The state space (X,B) is measurable. For the objective of the present paper 
it suffices to consider (q, u, v) of the state space. With a slight abuse of notation 
wc call \x\ := \\q\\ + \\u\\ + \\v\\ a norm on X. Further, let X be equipped with the 
natural induced topology then {x £ X : \x\ < k} is a compact subset of X for 
every k > 0. 

The underlying stochastic process X is piecewise deterministic. In particu- 
lar, X is a Borel right process and satisfies the strong Markov property, cf. [2], 
[6]. Thus, the process is equipped with the basic ingredients defining a strong 
Markov process. So, the process X is Borel right and is defined on a measurable 
space (^l,T) with values in the measurable space (X, B). Furthermore, the pro- 
cess is adapted to a filtration {Ft] and {P^, x £ X} are probability measures on 
(f2, T) such that for every x £\ we have ¥ X [X(0) = x] = 1. The collection 



defines a strong Markov family. The transition function P is defined by P(t, x, A) = 
f x [X(t) £ A] for all t > 0, x € X, and A £ B. 

For a set A £ B(E) let ta '■= inf{t > : X(t) £ A} denote the first entrance 
time, and for 5 > the first entrance time past 6 is given by ta{8) ■= inf{i > 
S : X(t) £ A}. By the Debut Theorem, cf. [14, Theorem A 5.1], the first 
entrance time defines a stopping time. Furthermore, for A £ B(E) we consider 




( 



O, T, {F t }, {X{t),t > 0}, {P,, x £ X}) 
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the occupation time tja, describing the number of visits by X to A, given by 



A Markov process X is called Harris recurrent if there exists a nontrivial o~- 
finite measure v such that whenever v(A) > and A G B(E) it holds that 
P x [ija = oo] = 1 for all x G E. If the invariant measure can be normalized to a 
probability measure, the Markov process X is called positive Harris recurrent. 
A multiclass queueing network is called stable if the underlying Markov process, 
denoted by X, is positive Harris recurrent. 

Beginning from the early 1990s with the works of Rybko and Stolyar [11], 
Stolyar [15] and Dai [6] an effective approach to investigate the stability of 
multiclass queueing networks is to consider the limits of scaled versions of the 
underlying Markov process. To this end, let (r„, cc„)„ e N be a sequence of pairs, 
where r n G M + and x n G X is a sequence of initial states. We assume that the 
sequence of pairs satisfies the following conditions 

lim r n — oo, limsup < oo, lim ^""^ = lira ^'"^ = 0, (1-1) 

where q n ,u n and v n denote the queue length, the residual intcrarrival time, 
and the residual service time, respectively. In the sequel, we consider the family 
X' := {X n (t) := ±X Xn (r n t) : t > 0,n G N} of Markov processes, where the 
superscript x n expresses the dependence on the initial state x n G X. Recall that, 
for each HL queueing network, (r„, x n )neN satisfying (1.1), and tu G G, there is 
a subsequence of pairs (r ni , x ni )i^jq such that 



uniformly on compact sets (u.o.c), that is, for each t > it holds that 



The set of all these limits is called the fluid limit model, denoted by TLM.. A 
fluid limit model of a queueing discipline is said to be stable if there is a r > 
such that for any fluid limit X(-) the Q(-) component satisfies Q(t) = for all 



Theorem (Dai [6]). Let a queueing discipline be fixed. Assume that As- 
sumptions (Al)-(AS) are satisfied. If the fluid limit model is stable, then the 
queueing network is stable. 

Furthermore, the work also shows that the fluid limits satisfy a set of equa- 
tions that is based on the first moments of the primitive increments. A com- 
pletely deterministic analog, called the associated fluid network, is obtained as 
the set of solutions to these equations. The associated fluid network is a superset 




lim T ±-X x "i{r ni t,u)=X(t,u) 



(1.2) 



lim sup 

i-s-oo 0<s <t 



^-X x ^(r nt s,Lu)-X(s,u)\ =0. 



t>r\\Q(0)\\. 
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to the fluid limit model and, thus, the stability analysis of multiclass queueing 
networks can be undertaken deterministicly. 

This result released a series of works establishing characterizing stability of 
the associated fluid networks, cf. [1], [3], [4], [5], [7], [8], [16]. In most cases the 
approach to verify the proposed stability conditions is by means of Lyapunov 
arguments. In [13] a consistent Lyapunov theory has been developed for fluid 
networks where the set of fluid level processes satisfies the following conditions: 
(a) Lipschitz continuity with respect to a global Lipschitz constant (b) scaling 
invariance (c) shift invariance (d) closedness in the topology of uniform con- 
vergence (e) a concatenation property holds (f) the set of fluid level processes 
depends lower semicontinuously on the initial fluid level. It is shown that sta- 
bility of a fluid network satisfying (a)-(f) is equivalent to the existence of a 
continuous Lyapunov function. 

In this paper we make use of the converse Lyapunov statement by explicitely 
using the Lyapunov function admitted by the stable fluid network to construct 
a Foster-Lyapunov function for the underlying Markov process. To this end, we 
prove a modified version of the Foster-Lyapunov theorem in [2, Proposition 4.5]. 
The line of argument is similar to/inspired by the one for Theorem 2 in [9]. 

Theorem 1.1. Let a queueing discipline be fixed and assume that the Assump- 
tions (Al)-(AS) hold. Suppose that the discipline is so that the associated fluid 
network satisfies (a) — (/). // the associated fluid network is stable, then the 
queueing network is stable. 

The remainder of the paper is structured as follows. In Section 2 we briefly 
recall known properties of fluid networks and display an equivalent Lyapunov 
characterization of stability of fluid networks satisfying (a) — (/). Section 3 is 
devoted to the proof of Theorem 1.1, where we explicitly use the Lyapunov 
function admitted by the stable associated fluid network to establish a Foster- 
Lyapunov function for the underlying Markov process. For this reason, we first 
prove an appropriate version of the Foster-Lyapunov theorem to conclude that 
the positive Harris recurrence of the underlying Markov process. 

2. Preliminaries on fluid networks 

In this section we provide an approach that was first considered by Rybko and 
Stolyar in 1992 [11] and was further developed by Stolyar [15] and Dai [6]. 

It is well-known that any fluid limit satisfies the following set of dynamic 
equations, where M = diag(/i): 



Q(t) = Q(0) + at + P T MT(t) - MT(t) > 0, 
T(0) = and T(-) is nondecreasing, 
W(t) = CM- 1 (Q(0) + A{t)) - CT(t), 
I(t) = et — CT(t) and /(•) is nondecreasing, 
I jit) can only increase when Wj(t) = 0, j G {1, J}, 



(2.1) 
(2.2) 
(2.3) 
(2.4) 
(2.5) 
(2.6) 



additional conditions on (Q(-),T(-)), specific to the discipline 
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Any pair (Q(-), T(-)) satisfying these equations is called a fluid solution. In ad- 
dition, the set of all fluid solutions to the equations (2.1)-(2.6) is called the 
associated fluid network, denoted by FN '. The associated fluid network is a 
purely deterministic network which is based on the mean values of the prim- 
itive increments of the stochastic queueing network. Further, as shown in [6], 
for any fluid limit X(-) the pair (Q(-),T(-)) is a fluid solution. An immediate 
consequence of the above theorem is that FLM C J-M. Note that, in general 
the inclusion is strict, sec [7, Section 2.7]. Furthermore, the fluid solutions are 
not unique in general. A related counterexample can be found in [2] Example I 
in Section 4.3. 

Let Q denote the set of all fluid level processes Q(-) such that there is an 
allocation process T(-) such that the pair (Q(-),T(-)) is a fluid solution. An 
associated fluid network is called stable if there exists a r > such that Q(t) =0 
for all t > t||Q(0)|| and for all Q £ Q. In order to state generic properties of 
the set of fluid level processes Q we introduce a scaling and a shift operator. 
Given a function / : R + — > R^f, for r > the scaling operator a r is defined by 
°V / (t) -fi^t), an d for s > the shift operator S s is defined by S s f (t) := 
f(t + s). The subsequent proposition summarizes well-known generic properties 
of fluid networks. 

Proposition 2.1. The set Q is nonempty and satisfies 
(a) there is a L > 0, such that for any Q £ Q and t, s £ R+ it holds that 



(b) Q £ Q implies ay Q £ Q for all r > 0. 

(c) Q £ Q implies 8 S Q £ Q for all s > 0. 

(d) if a sequence (Q n )nen in Q converges to Q* u.o.c, then Q* £ Q. 

Let Qi and Qi be two fluid level processes in Q satisfying Qi(t*) = Qa(0) 
for some t* > 0. Then, Qi o t » Q2 is called the concatenation of Q\ and Q2 at 
t* , which is defined by 



The set Q of fluid level processes is said to satisfy the concatenation property if 
for any Qi and Q 2 in Q such that Qi(t*) = Q2(0) for some t* > it holds that 
Qi Of Q-2 € Q. Moreover, let Q q := {Q £ Q : Q(0) = q} denote the set of fluid 
level processes in Q starting in q £ R™ . Further, the set- valued map q ~-> Q q 
is lower semicontinuous if for each q £ R^, Q £ Q q and (q n )n£N converging 
to q, there is a sequence of fluid level processes (Q n )neN with Q n £ Q qn which 
converges to Q u.o.c. 

Remark 2.1. In [12] it is shown that for fluid networks under general work- 
conserving, priority and HL proportional processor sharing disciplines the set Q 
of fluid level processes has the properties (a) — (d) and additionally 



\\Q(t)-Q(s)\\<L\t-s\. 




t<t*, 
t > t*. 
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(e) it satisfies the concatenation property, 

(/) the set-valued map q Q q is lower semicontinuous. 

We note that FIFO fluid networks the concatenation property does not hold in 
general, cf. Example 5.2.15 in [12]. 

To characterize stability of associated fluid networks let K, denote the set of 
all functions w : R+ — > R+ being strictly increasing with w(0) = 0. Further, let 
/Coo denote all unbounded class K functions. Given an associated fluid network, 
a function V : R+ — > R+ is said to be a Lyapunov function if there exist class 
/C functions Wi, i = 1, 2, 3 such that 



for all < s < t G R+ and all trajectories Q G Q. A related definition was given 
by Dai, see [7]. 

Theorem 2.1 ([13]). An associated fluid network satisfying (a) — (/) is stable 
if and only if it admits a continuous Lyapunov function and the functions Wi, 
i = l,2,3 are of class /Coo ■ 

3. Proof of the Theorem 1.1 

In this section we exhibit how the converse Lyapunov Theorem 2.1 can be used 
to give a new proof of fluid approximation theorem if the associated fluid net- 
work satisfies the conditions (a) — (/). Based on the Lyapunov function of the 
associated fluid network wc define a Foster-Lyapunov function for the under- 
lying Markov process of the HL queueing network. To this end, we prove a 
modified version of the Foster-Lyapunov theorem which is appropriate for our 
purpose. Then, we apply the modified Foster-Lyapunov Theorem to conclude 
the positive Harris recurrence of the underlying Markov process. 

Let a be a probability measure on (0, oo) and consider the Markov process 
X a with transition function 



where x G X and A G B. Let fj, be some nontrivial measure on (E, 13(E)). A 
nonempty set A G B is called petite if there is a nontrivial measure fj, on (X, B) 
and a probability measure a on (0, oo) such that the transition function T a (x, B) 
of the sample process satisfies T a (x, B) > n(B) for all x G A and for all B G B. 
The subsequent modification of the Foster-Lyapunov criterion for positive Harris 
recurrence is close to Proposition 4.5 in [2]. 

Theorem 3.1. Suppose that X is continuous time Markov process, such that 
there exist e > 0, n > 0, c > 0, and measurable function W : X — > R with 



w 1 (\\q\\)<V(q)<w 2 (\\q\\), 



(2.7) 




(2.8) 
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W{x) >5>0 and 

E x [W(X(cW(x)))} < max{VF(x),K} -sW(x) (3.1) 

for all x G X. Then, for all x, 

E x [t b (S)) < lmax{W(x), K }, 

where B := {x G X : W(x) < n}. In particular, if B is a closed petite set, X is 
positive Harris recurrent. 

Proof. Let n > 0, e > and c > and let (T n ) n6N denote a sequence of stopping- 
times defined by To := and 

T n+1 :=T n + cW(X(T n )). (3.2) 

We abbreviately denote by T n := Tr n the cr-algebra corresponding to the stop- 
ping time T n . The strong Markov property and condition (3.1) imply that 

E x [W(X(T n )) | F n - X ] = E x [ W(X(T n ^ + cW(X(T n ^) ) ) )|.F n _i] 
= E Jf(T „_ l) [ty(X(cW(X(T„_ 1 ) ) ) )] 

< max{W(X(T n _ 1 )), K } - eW{X{T n _ x )). (3.3) 

Further, let M(0) := max{W(x), k} and for n > 1 we define 

M(n) := cW{X(T n j) + eT n . (3.4) 

Also, we note that T n G J- n -i- 

Next, we show that for AT = inf{?i G N : M(n) G B} and for all n < JV we 
have 

E x [M(n)| J n _i] <M(n-l). 

To see this, first note that for n < N it holds that m.&x{W(X(T n -i)), k} = 
W{X{T n ^)) as well as T n G J"„_i. Moreover, using (3.2) and (3.3) it holds 
that 

E x [M(n) | T n -x ] = E x [cW(X(T n )) + sT n \ T n - X ] 

= cE x [W(X(T n )) | T n -x ] + eE x [T„ | T n - X ] 
< cmax{l¥(A(T n _ 1 )), K } - eW{X{T n _ x )) + eE x [T n \ T n _ x } 

= eW(X(T n -i)) - e(T„ - T n _ x ) + eE x [T„ \ T n - X } = M(n - 1). 

The validity of the last equality follows from the basic properties of stopping 
times and expectations. Hence, M(min{n,iV}) is a supermartingalc on T n . 
Moreover, by the Optional Sampling Theorem we have that 



E X [M(N)} <E x [M(0)] = max{W(x),K}. (3.5) 
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Besides, since W(X(T n )) < M{n) it also holds that tb(S) < T/v and together 
with (3.4), (3.5) it follows that for all x € X we have 

eE x [t b (S)] < E„.[M(J\0] < m&x{W(x), K } 

and, hence, we have P x [tb < oo] = 1 for all x <E X and 

supE x [r B (<5)] < f. 

x£B 

The assertion then follows from Theorem 4.1 in [2] (or Theorem 1.1, 1.2 in 
[10]). □ 

Proof of Theorem 1.1. Since TChA C TM and the associated fluid network is 
stable, there is a r > such that for all Q € TLM. it holds that Q(t) = 
for all t > t||Q(0)||. Also, from the converse Lyapunov Theorem 2.1 there is a 
continuous Lyapunov function Vl and class ICqq functions Wi, i = 1,2,3 such 
that for q £ we have that 

W 1 (\\q\\)<V L (q)<W 2 (\\q\\) 
MQ{t)) < -w 3 (\\Q(t)\\). 

Further, let (r n ,x n ) n ^ be a sequence of pairs satisfying (1.1). Then, along a 
subsequence, which is also indexed by n, it holds that 

±Q Xn (rnt) -+Q(t) u.o.c. 

as n — > oo. In particular, the stability of the fluid limit model implies that 

±Q Xn {r n r) ->0. 

That is, for any e £ (0, 1) there is a N £ N such that for all n > N we have 

±w^(V L (Q^(r n T)))<e. 

Moreover, since (Vi(Q x "(r n i))) < HQ 31 " ( r n-*) II an d by Lemma 4.5 in [6] we 
have that 

{^w^(V L (Q^(r n r))),n€N} 
is uniformly integrablc. In addition, it also holds that 

lim J-||C7*«(r„r)|| = lim J-\\V x -(r n r)\\ = 0. 

n—too n n— too n 

The families |^°(r„T), n £ n\ and j^-V^" (r„r), n £ n| are for each k G 
{1, A'} uniformly integrablc by Lemma 4.3 in [6]. Hence, we have 

limsup^E^-^Q^r))) + ||E£"(r n r)|| + \\V^(r n r)\\] < e. 
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Thus, there is a k > such that for all r n > k we have 

E [w^(V L (Q x -(r n r))) + \\ U x ~(r n r) \\ + || V x -{r n r) || ] < r n e. 
We define the Foster-Lyapunov function W : X — >■ R+ by 

:=«; 2 - 1 (T/ L ( (7 )) + || U || + ||«||. 
Then, for all x with W(x) > k it follows that 

E a .[W(X(W(a:)T))] < £W^(a;) 

and consequently, 

E x [W{X{tW{x))) \ < m&x{W{x),n}-eW(x). 

Finally, we have to show that B = {x : W(x) < k} is closed and petite. To see 
that B is petite, note that by Lemma 3.1 in [6] the set A = {x G X : |x| < k} 
is closed and petite. Furthermore, since w^ 1 (Vl(<7)) < ||g|| it holds that B C A. 
In addition, the continuity of w^ 1 and Vl imply that B is closed. As subsets of 
petite sets are petite, the assertion then follows from Theorem 3.1. □ 
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